Method and Device for Displaying Vehicle Surroundings

ABSTRACT

The invention relates to a device (100) as well as a method for displaying vehicle surroundings in a vehicle (1) during an instantaneous second time point (t). The method comprises the steps:providing (S1) a first blind area image (I_B[t]), which contains an image synthesis of a blind area (B) of the vehicle surroundings, which blind area is arranged outside of a field of view (141F, 141R, 141LL, 141LR) of a vehicle camera (140F, 140R, 140LL, 140LR), at a first time point (t−1) preceding the second time point (t),arranging (S2) each first blind area pixel (IB_SP[t−1]) of the blind area (B) in a new position estimated for the second time point (t),determining (S3) whether the new position of each first blind area pixel (IB_SP[t−1]) at the second time point (t) still lies within the blind area (B), andproducing (S4) a respective second blind area pixel (IB_SP[t]) for the second time point (t) by synthesizing (S5A) each first blind area pixel (IB_SP[t−1]) on the basis of motion data (v_x, v_y) of the vehicle (1) if the new position is determined to lie within the blind area (B).

The invention relates to a method and a device for displaying vehiclesurroundings in a vehicle during an instantaneous time point.

A vehicle can have a driving assistance system which displays orrespectively reproduces the vehicle surroundings for a vehicle passengerin the interior of the vehicle on a display apparatus, such as a screen.Such a driving assistance system can also be referred to as a surroundview system and usually has one or more vehicle cameras which aremounted on the vehicle and have different fields of view or respectivelymake possible viewing angles.

Although multiple fields of view are detected by the different vehiclecameras, there remains at least one blind area, also referred to as ablind spot, which lies outside of all of the fields of view and istherefore not contained in the camera images. An example of such a blindarea is a ground level which is being driven over by the vehicle in theobserved moment and is thus concealed. This is because the variousvehicle cameras are usually arranged e.g. on the vehicle front, the rearof the vehicle and the vehicle sides and look out from there, such thatthe ground level is constantly concealed by the vehicle itself orrespectively its body. Therefore, this region is mostly omitted in thedisplay and occupied, for example, by a place holder. Not only is thisplace holder visually unappealing, but it can also be irritating for thevehicle passenger.

It is therefore an object of the invention to make it possible toimprove the way the vehicle surroundings are displayed in a vehicle.

This object is achieved by the subject-matter of the independent claims.Advantageous embodiments and further developments of the invention areindicated in the dependent claims, the description and the accompanyingfigures.

A first aspect of the invention relates to a method for displayingvehicle surroundings in a vehicle during an instantaneous second timepoint. The instantaneous time point can be understood to mean that thevehicle surroundings are substantially displayed in real time during thevehicle operation. The vehicle can move in any direction at a vehiclespeed which can be determined e.g. from odometry data, wherein the realtime display is effected with a suitable image refresh rate.

The method according to the invention provides the following steps:

-   -   A first blind area image, e.g. a synthesized or respectively        rendered image, having a plurality of blind area pixels is first        provided, which contains an image synthesis, e.g. a rendering,        of a blind area of the vehicle surroundings, which blind area is        arranged outside of a field of view of a vehicle camera, at a        first time point preceding the second time point. That is to say        that the first time point therefore lies temporally before the        observed instantaneous second time point.    -   Each first blind area pixel of the blind area is then arranged        in, e.g. transformed into, a new position estimated, e.g.        predicted, for the second time point. This means that the blind        area image has a multiplicity of blind area pixels which are        arranged from their old position, as contained in the blind area        image, in their new position based on an estimation, prediction,        transformation, interpolation or similar. Instead of a        respective individual blind area pixel, a combination of blind        area pixels can also be enlisted, in order to thereby reduce any        aliasing effects.    -   It is then determined whether the new position of each first        blind area pixel at the second time point still lies within the        blind area. Due to the vehicle motion, the observed, newly        arranged blind area pixel could now also lie outside of the        blind area.    -   A respective second blind area pixel, e.g. a rendered pixel, is        then produced, e.g. synthesized, rendered, etc. for the second        time point by synthesizing each first blind area pixel on the        basis of motion data of the vehicle in a motion-compensated form        if the new position is determined to lie within the blind area.        Since the vehicle can continue moving from blind area image to        blind area image as it is driving, the blind area pixel can        likewise alter its position so that the motion data of the        vehicle are taken into consideration in order to produce the        second blind area pixel. If the vehicle is not moving, the        second blind area pixel can correspond to the first. The motion        data can be obtained e.g. from the odometry of a vehicle system        or from a motion estimate from multiple camera images.

In the same manner, the proposed method offers multiple advantages withthis configuration. Thus, the blind area of the vehicle surroundings istemporally reconstructed, which manages without a place holder whichwould permanently overlay the blind area regardless of the vehiclesurroundings to be displayed. In addition, due to the method which is inprinciple recursive, only a single first blind area image can beretained and provided so that significant storage resources can be savedin e.g. a storage unit and/or a data processing unit. Thus, the costs ofproviding a driving assistance system operated in accordance with thismethod can be reduced.

An advantageous further development provides that in the event that thedetermined new position of each first blind area pixel is determined tolie outside of the blind area, the respective second blind area pixelcan be produced for the second time point by synthesizing each firstblind area pixel on the basis of at least one captured camera imagewhich is provided at the first time point. That is to say that, in thiscase, an already synthesized, e.g. rendered, image no longer serves as abasis for the further image synthesis, e.g. rendering, but instead acaptured camera image of the vehicle surroundings. This likewisesignificantly saves storage resources since only a single camera imagehas to be retained and provided. If the vehicle has multiple vehiclecameras, e.g. one having a viewing direction from the vehicle front, onehaving a viewing direction from the rear of the vehicle and one eachhaving a viewing direction from each lateral vehicle side, each vehiclecamera can in each case provide its own camera image.

In a further embodiment, it can be determined on the basis of a motionvector obtained from the motion data of each first blind area pixel,from which individual vehicle camera from a multiplicity of differentlyarranged and/or aligned vehicle cameras, which each have differentfields of view, the camera image used for the pixel synthesis isprovided. Thus, a first blind area pixel which is moving into the fieldof view of a vehicle camera looking ahead of the vehicle, due to thevehicle motion, can, for example, be produced or respectivelysynthesized, e.g. rendered, on the basis of the one camera image of thisvehicle camera.

Depending on the direction of motion of the vehicle, that is to saytravelling straight ahead, reversing and/or cornering, the motion vectorcan be arranged in the respective field of view of a correspondingvehicle camera. That camera image of the respective camera, in the fieldof view of which the motion vector is arranged, can be selected to beprovided.

In order to significantly save storage resources, it is advantageous ifexactly one camera image or respectively an individual frame perexisting vehicle camera is retained to be provided.

Regarding this, it can also be advantageous if exactly one blind areaimage for the first time point is retained to be provided.

In order to be able to reproduce a display image of the vehiclesurroundings which is to be displayed e.g. on a screen of the vehicle, asecond blind area image produced or respectively synthesized, e.g.rendered, from the respective second blind area pixels for the secondtime point can be combined with at least one instantaneous camera imageof the second time point. In this case, the fields of view are thereforerepresented by real-time camera images and the blind area is representedby the second blind area image for the instantaneous second time point.

In particular during the initial start of a device and/or drivingassistance system operated in accordance with this method, the situationcan occur that no first blind area image of the blind area yet exists.In order to hide the blind area, a display image to be displayed in theinterior of the vehicle can then be displaced into the field of view ofthe respective vehicle camera, e.g. the front camera.

The method can be particularly advantageously deployed if the blind areais a ground level which is being driven on by the vehicle orrespectively which is concealed by the vehicle body. Instead of a placeholder, a synthesized, e.g. rendered, display image can then bedisplayed.

The method described above can be implemented e.g. in a controlapparatus such as a control unit of the vehicle. The control apparatuscan have a storage apparatus and a data processing apparatus.

A further aspect of the invention relates to a device for displayingvehicle surroundings in a vehicle during an instantaneous second timepoint. The device has:

-   -   At least one vehicle camera for detecting a corresponding field        of view of the vehicle surroundings.    -   A storage apparatus, in which exactly one first blind area image        is retained, which contains an image synthesis of a blind area        of the vehicle surroundings, which blind area is arranged        outside of the field of view, at a first time point preceding        the second time point.    -   A data processing apparatus which is designed i) to arrange each        first blind area pixel of the blind area in a new position        estimated for the second time point, ii) to determine whether        the new position of each first blind area pixel at the second        time point still lies within the blind area, and iii) to produce        a respective second blind area pixel for the second time point        by motion compensating each first blind area pixel on the basis        of motion data of the vehicle if the new position is determined        to lie within the blind area.

The device can be further developed in accordance with the methoddescribed above and in particular offers the advantage that due to theprocedure, which is in principle recursive, storage resources can besignificantly saved.

The invention is in particular suitable as a driving assistance systemfor a vehicle, having a display apparatus arranged in the interior ofthe vehicle, such as e.g. a screen.

Advantageous embodiments of the invention are explained in greaterdetail below, with reference to the appended figures, wherein:

FIG. 1 shows a vehicle having a driving assistance system according toan aspect, which has a device for displaying vehicle surroundingsaccording to a further aspect of the invention,

FIG. 2A shows a diagram of the production of a blind area image, whereina previous, synthesized blind area image serves as the basis,

FIG. 2B shows a diagram of the production of a blind area image, whereina previous, captured camera image serves as the basis,

FIG. 2C shows a diagram of the production of a blind area image, whereina previous, captured camera image serves as the basis, and

FIG. 3 show a flowchart of a method according to a further aspect of theinvention.

The figures are merely schematic representations and only serve toexplain the invention. The same or similarly acting elements areconsistently provided with the same reference numerals.

FIG. 1 shows a vehicle 1 standing on a ground level, which cansubstantially move in the directions x and y and which has a device 100in the form of a driving assistance system or respectively a surroundview system. This makes it possible for a vehicle passenger to have apanoramic view of the instantaneous vehicle surroundings displayed tohim in the interior of the vehicle as a display or respectivelypictorial reproduction.

To this end, the device 100 has a display apparatus 110 in the form of ascreen, which is arranged in the interior of the vehicle 1 and visuallyrepresents a display image I. Moreover, the device 100 has a dataprocessing apparatus 120 having at least one processor (which is notdescribed in greater detail), which interacts with the display device110, and a storage apparatus 130 which likewise interacts therewith. Inaddition, the device 100 has a multiplicity of vehicle cameras 140F,140R, 140LL, 140LR which are mounted at different positions of thevehicle 1 and have different viewing angles or respectively fields ofview 141F, 141R, 141LL, 141LR. Thus, the vehicle camera 140F is arrangedon the vehicle front, the vehicle camera 140R is arranged on the rear ofthe vehicle, the vehicle camera 140LL is arranged laterally on the leftand the vehicle camera 140LR is arranged laterally on the right. Thefields of view 141F, 141R, 141LL, 141LR can be detected as therespective camera image I_F, I_R, I_LL, I_LR and are directly reproducedon the display apparatus 110 and, if applicable, are (temporarily)stored in the storage apparatus 130. For displaying, the camera imagesI_F, I_R, I_LL, I_LR are composed or respectively combined by the dataprocessing apparatus 120 into the display image I (see FIGS. 2A-2C).

It is obvious from FIG. 1 that in the ground level, due to the vehicle 1located thereon or respectively the body thereof, a blind area B isproduced, which does not lie in any of the fields of view 141F, 141R,141LL, 141LR and which cannot therefore be detected by any of thevehicle cameras 140F, 140R, 140LL, 140LR, since the body obscures therespective view. Consequently, no direct camera image can be reproducedfor the blind area B as a blind area image I_B of the display image I onthe display apparatus 110.

In order to nevertheless represent the blind area B with a substantiallyphoto-realistic display in the form of the blind area image I_B on thedisplay apparatus 110, the device 100 can be operated with the methoddescribed below on the basis of FIGS. 2A-2C and 3.

In principle, the vehicle surroundings are to be displayed on thedisplay apparatus 110 substantially in real time at an instantaneoustime point t, that is to say the display image I is to contain thevehicle surroundings at the instantaneous time point t. This isindicated below as well as in FIGS. 2A-2C by a time reference [t] in thereference numerals. Accordingly, a first time point t−1 preceding theinstantaneous second time point t is referenced in FIGS. 2A-2C with[t−1].

When the device 100 is initially started, the blind area B is hidden inthe display image I of the display apparatus 110 in an optional step S0,by displacing the display image I of the vehicle surroundings so farinto one of the fields of view 141F, 141R, 141LL, 141LR of the vehiclecameras 140F, 140R, 140LL, 140LR that the blind area B is thus hidden.The respective field of view 141F, 141R, 141LL, 141LR can be selectede.g. as a function of the motion direction x, y of the vehicle, themotion data thereof such as a vehicle speed v_x, v_y, the steeringangle, the gear selection of the transmission, etc. To put it moresimply, the display image I runs ahead of the actual vehiclesurroundings at time point t e.g. by a vehicle length so that the blindarea B is hidden. An initial blind area image I_B[t−1] can thus bestored and retained, which, instead of the blind area B, contains animage section of the respective camera image IF[t−1], IR[t−1], ILL[t−1],ILR[t−1].

In a step S1, the blind area image I_B[t−1] of the time point t−1 isthen provided at the instantaneous time point t from the storageapparatus 130, which blind area image only contains the image contentexplained above in the first run-through following the initial start ofthe device 100, but which is constantly updated during operation asdescribed below. If the vehicle 1 continues moving at e.g. 15 m/s in thex-direction, the image content of the blind area image I_B is updatedwith a corresponding image refresh rate and contains the image contentat the preceding time point t−1 when it is provided for the respectiveinstantaneous time point t.

In a step S2, each first blind area pixel IB_SP[t−1] of the blind areaimage I_B[t−1] is arranged by the data processing apparatus 120 in a newposition estimated or respectively predicted for the instantaneous timepoint t by e.g. a transformation or similar, for which reason the motiondata v_x, v_y of the vehicle 1 are in particular considered. These aremade available e.g. by a vehicle system, e.g. a vehicle bus, or aredetermined from the camera images I_F, I_R, I_LL, I_LR. The new positioncan be estimated or respectively predicted e.g. by a suitable imageprocessing method. If the vehicle 1 has continued moving at e.g. v_x=15m/s in the x-direction, a corresponding motion vector SP_V is determinedfor each first blind area pixel IB_SP[t−1] and this is accordinglyarranged as a second blind area pixel IB_SP[t] in its new position, asindicated in FIG. 2A.

It is then determined in a step S3 by the data processing apparatus 120whether the new position of the respective blind area pixel IB_SP at theinstantaneous time point t still lies within the blind area B. In theexemplary embodiment shown in FIG. 2A, a respective blind area pixelIB_SP shifted or respectively transformed by the motion vector SP_Vstill lies within the blind area B.

In this case, a respective second blind area pixel IB_SP[t] is produced,e.g. rendered, for the instantaneous time point t in the data processingapparatus 120, in a step S4, by motion compensating each first blindarea pixel IB_SP[t−1], on the basis of the motion data of the vehicle 1in a step S5A, i.e. in particular shifting it by the motion vector SP_V.As indicated in FIG. 2A for two exemplary blind area pixels IB_SP, thisis repeated frequently until such time as each first blind area pixelIB_SP [t−1] is preferably arranged in its new position and a secondblind area image I_B[t] is produced, e.g. rendered, therefrom, and isdisplayed in the display image I [t] in place of the blind area B. Thesecond blind area image I_B[t] is then stored in the storage apparatus130 and is provided in the next run-through of the method beginning withstep S1 as a first blind area image I_B[t−1], which has been updated interms of its content, for a further second time point t+n.

This can be generalized as follows:

I_B[x, y, t]=I_B[x+v_x, y+v_y, t−1],

wherein I_B is the blind area image, v_x and v_y are the motion data inthe x- or respectively y-direction, t−1 is the first time point and t isthe instantaneous second time point.

FIG. 2B shows another possible case of the determination in step S3, inwhich the new position of each first blind area pixel IB_SP[t−1] at theinstantaneous time point t no longer lies within, but instead outside ofthe blind area B. Thus, in the exemplary embodiment shown in FIG. 2B,the respective blind area pixel IB_SP which has been shifted orrespectively transformed by the motion vector SP_V lies in the field ofview 141F of the vehicle camera 140F due to the motion of the vehicle inthe x-direction.

In this case, the respective second blind area pixel IB_SP[t] for theinstantaneous time point t is produced, e.g. rendered, in step S4, bysynthesizing, e.g. rendering, each first blind area pixel IB_SP[t] onthe basis of the captured camera image I_F[t−1] which is provided at thefirst time point t−1, in a step S5B.

FIG. 2C shows this case of the determination in step S3 using theexample of cornering to the right, which comprises a motion both in thex- and y-directions having the motion data v_x, v_y. As shown in FIG.2C, the respective blind area pixel IB_SP which is shifted orrespectively transformed by the motion vector SP_V now lies in the fieldof view 141LR of the vehicle camera 140LR, due to the motion of thevehicle in the x-direction.

In this case, the respective second blind area pixel IB_SP[t] isproduced, e.g. rendered, for the instantaneous time point t in step S4,by synthesizing, e.g. rendering, each first blind area pixel IB[t−1] onthe basis of the captured camera image I_LR[t−1] which is provided atthe first time point t−1 in step S5B. As indicated in FIG. 2B for twoexemplary blind area pixels IB_SP, this is repeated frequently untilsuch time as each first blind area pixel IB_SP [t−1] is preferablyarranged in its new position and a second blind area image I_B[t] isproduced, e.g. rendered, therefrom, and is displayed in the displayimage I [t] in place of the blind area B. The second blind area imageI_B[t] is then stored in the storage apparatus 130 and provided in thenext run-through of the method beginning with step S1 as a first blindarea image I_B[t−1] which has been updated in terms of its content for afurther second time point t+n.

In steps S5B according to FIGS. 2B and 2C, the blind area pixel IB_SP[t]to be displayed is accordingly produced by an image synthesis, e.g.rendering, of a captured camera image I_F[t−1] or respectively I_LR[t−1]at time point t−1. This principle can of course be easily transferred tothe remaining camera images I_R and I_LL.

This can be generalized as follows:

I_B[x, y, t]=I_F, I_R, I_LL, I_LR[x+v_x, y+v_y, t−1],

wherein I_B is the blind area image, I_F-I_LR are the camera images ofthe vehicle cameras 140E-140LR, v_x and v_y are the motion data in thex- or respectively y-direction, t−1 is the first time point and t is theinstantaneous second time point.

In an optional step S6, the display image I for the display apparatus110 is then combined from the second blind area image I_B[t] and theinstantaneous camera images I_F[t], I_R[t], I_LL[t], I_LR[t] of thesecond time point t.

FIG. 3 shows the method described above having the optional step S0, thesteps S1-S5 as well as the optional step S6, which is once againsummarized in a flowchart.

1. A method for displaying vehicle surroundings in a vehicle (1) duringan instantaneous second time point (t), having the steps: providing (S1)a first blind area image (I_B[t]), which contains an image synthesis ofa blind area (B) of the vehicle surroundings, which blind area isarranged outside of a field of view (141F, 141R, 141LL, 141LR) of avehicle camera (140F, 140R, 140LL, 140LR), at a first time point (t−1)preceding the second time point (t), arranging (S2) each first blindarea pixel (IB _SP[t−1]) of the blind area (B) in a new positionestimated for the second time point (t), determining (S3) whether thenew position of each first blind area pixel (IB_SP[t−1]) at the secondtime point (t) still lies within the blind area (B), and producing (S4)a respective second blind area pixel (IB_SP [t]) for the second timepoint (t) by synthesizing (S5A) each first blind area pixel (IB_SP[t−1]) on the basis of motion data (v_x, v_y) of the vehicle (1) ifthe new position is determined to lie within the blind area (B).
 2. Themethod according to claim 1, wherein the respective second blind areapixel (IB_SP[t]) is produced (S4) for the second time point (t) bysynthesizing (S5B) each first blind area pixel (IB_SP[t−1]) on the basisof at least one captured camera image (I_F[t−1], I_R[t−1], I_LL[t−1], I_LR[t−1]) which is provided at the first time point (t−1) if the newposition is determined to lie outside of the blind area (B).
 3. Themethod according to claim 2, wherein it is determined on the basis of amotion vector (SP_V) obtained from the motion data (v_x, v_y) of eachfirst blind area pixel (IB_SP[t−1]), from which vehicle camera from amultiplicity of vehicle cameras (140F, 140R, 140LL, 140LR), which eachhave different fields of view (141F, 141R, 141LL, 141LR) , the cameraimage (I_F[t−1], I_R[t−1], I_LL[t−1], I_LR[t−1]) is provided.
 4. Themethod according to claim 3, wherein the camera image (I_F[t−1],I_R[t−1], I_LR[t−1], I_RR[t−1]) of the respective camera (140F, 140R,140LL, 140LR), in the field of view (141F, 141R, 141LL, 141LR) of whichthe motion vector (SP_V) is arranged, is selected to be provided.
 5. Themethod according to claim 3, wherein exactly one camera image (I_F[t−1],I_R[t−1], I_LR[t−1], I_RR[t−1]) per vehicle camera (140F, 140R, 140LL,140LR) is retained to be provided.
 6. The method according to claim 1,wherein exactly one blind area image (IB_SP[t−1]) of the first timepoint (t−1) is retained to be provided.
 7. The method according to claim1, wherein a second blind area image (IB_SP[t]) produced from therespective second blind area pixels (IB_SP[t]) for the second time point(t) and at least one instantaneous camera image (I_F[t], I_R[t],I_LL[t], I_LR[t]) of the second time point (t) are combined fordisplaying in the vehicle (1).
 8. The method according to claim 1,wherein at the initial start a display image (I) of the vehiclesurroundings is displaced so far into the field of view (141F, 141R,141LL, 141LR) that the blind area (B) is hidden.
 9. The method accordingto claim 1, wherein the blind area (B) is arranged in a ground levelwhich is driven over by the vehicle (1).
 10. A device (100) fordisplaying vehicle surroundings in a vehicle (1) during an instantaneoussecond time point (t), having at least one vehicle camera (140F, 140R,140LL, 140LR) for detecting a corresponding field of view (141F, 141R,141LL, 141LR) of the vehicle surroundings, a storage apparatus (130), inwhich exactly one first blind area image (I_B[t−1]) is retained, whichcontains an image synthesis of a blind area (B) of the vehiclesurroundings, which blind area is arranged outside of the field of view(141F, 141R, 141LL, 141LR), at a first time point (t−1) preceding thesecond time point (t), and a data processing apparatus (120) which isdesigned i) to arrange each first blind area pixel (IB_SP[t−1]) of theblind area (B) in a new position estimated for the second time point(t), ii) to determine whether the new position of each first blind areapixel (IB_SP[t−1]) at the second time point (t) still lies within theblind area (B), and iii) to produce a respective second blind area pixel(IB_SP[t]) for the second time point (t) by synthesizing (S5) each firstblind area pixel (IB_SP[t−1]) on the basis of motion data (v_x, v_y) ofthe vehicle (1) if the new position is determined to lie within theblind area (B).